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What We Studied 


Students living in rural areas are less likely to attend schools that offer computer science (CS) courses largely because 
educational institutions in these remote areas lack the resources to staff teaching positions for these courses. This study 
investigated the impact of WeTeach_CS, a program designed to train teachers to become certified to teach high school 
CS in Texas. The WeTeach_CS collective impact model may be well suited to influence rural areas at scale because it 
utilizes an existing network of organizations across the state to bring high-quality professional development 
opportunities to teachers in remote areas. Results from an interrupted time series analysis showed a significant, positive 
change in the rate in which the number of certified CS teachers in rural areas increased during the period of time after 
WeTeach_CS began compared to the period before the program was implemented, whereas the number of teachers 
certified in technology applications showed no such change. Furthermore, the growth rate in the number of certified CS 
teachers was much higher for rural schools than urban, suggesting that collective impact models like WeTeach_CS 
may be especially beneficial for rural communities. 


Schools in rural areas are behind their more urbanized counterparts in terms of access to CS education (Google Inc. & 
Gallup Inc, 2017). Rural schools constituted only 10% all U.S. public schools that offered an AP computer science 
course in 2017 (College Board, 2018), even though the latest estimates show that 28% of all public schools in the U.S. 
are designated as rural and 42% as rural or small-town (National Center for Education Statistics, 2014). At the core of 
this issue is that the lack of qualified CS teachers overall is exacerbated in rural areas, where schools have a relatively 
harder time funding new teaching positions and recruiting highly qualified teachers to fill those positions. Schools 
simply cannot offer courses for which they have no qualified teacher, implying that one crucial antecedent to 
increasing access to CS education for underserved students is increasing the CS teacher workforce in those areas. 


Efforts to increase the number of qualified CS teachers are invariably met with challenges of scalability and 
sustainability. Collective impact models (Kania & Kramer, 2011), which unite interdisciplinary partners towards a 
common goal, may be better suited to overcome these challenges than stand-alone interventions. A collective impact 
model has the potential to effect change in rural schools because it connects local education agencies with resources 
and expertise to which they may not otherwise have access. The WeTeach_CS program has worked extensively with 
rural schools and teachers over the past three years to build capacity for K-12 CS Education. Approximately 40% of 
the 640 schools served by WeTeach_CS in 2016-17 are in rural or small-town districts. WeTeach_CS, which serves 
teachers across multiple districts, is particularly helpful in creating an economy of scale for CS teacher professional 
development that is not possible to obtain for individual small, rural districts. WeTeach_CS supports sustainability of 
program effects by using a network of professional development partners to support local teachers before and after 
becoming certified to teach CS. 
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How We Analyzed the Data 


This research project examines the effectiveness of the WeTeach_CS collective impact model for increasing the 
number of certified CS teachers in rural schools. Interrupted times series (ITS) analysis was used to investigate the 
impact of the WeTeach_CS program on the number of certified computer science teachers in rural schools. ITS is a 
quasi-experimental method that tests whether the introduction of an intervention produces a change in the target 
outcome over time for the treatment group to a greater degree than a comparison group. We refer to the periods of time 
before and after the introduction of the program as the pre-intervention and intervention phases, respectively. Two 
types of causal effects can be estimated with ITS: level change and slope change. In the current analysis, the presence 
of a level change would mean a change in the number of certified teachers immediately following the introduction of 
the WeTeach_CS program that could not be explained by the overall trend. A statistically significant slope change 
would indicate that the intervention phase trend differs from the pre-intervention phase trend. Similar level and slope 
changes for the comparison group would suggest that the observed changes for the treatment group were due to some 
other factor not accounted for in the analysis. 


We hypothesized that the WeTeach_CS program would positively impact the number of certified CS teachers in rural 
schools but that this effect would happen gradually over time. Accordingly, we expected results to reveal a statistically 
significant, positive change in slope but no significant level change. To help rule out the possibility of unknown 
confounders, we included teachers who obtained certification in technology applications to serve as the comparison 
group. Technology applications is the certification most similar to the computer science licensure in terms of the 
content knowledge covered on the certification exam and the curricular content of the courses that a teacher holding 
that certification would be qualified to teach. Null results for changes in the level and slope of the number of certified 
technology applications teachers would support inferences that any changes in the number CS-certified teachers are 
attributable to the WeTeach_CS program. We included data on the number of certified teacher across six years, from 
the beginning of the 2011-12 school year through the end of the 2016-17 school year. 


What We Discovered 


Results showed that, as expected, the level change was not statistically significant, indicating that there was no 
immediate change in the number of certified CS teachers that could not be explained by the overall trend (see Table 1). 
However, the slope change was statistically significant, signaling that the intervention slope differed from the pre- 
intervention slope. The coefficient for this estimate (B=6.46) suggests that an average of approximately six more 
teachers were certified each month in the 


intervention phase than in the pre-intervention Table 1. Results of the interrupted time series analysis for 
phase. By adding the slope-change coefficient jneichercaeetited inccenmiee eieene 

to the coefficient for the preintervention slope <7 SE , ? 
(B=0.33), we can calculate the actual slope for the SS 
intervention phase. Pre-intervention slope 0.33 015 294 .037 
In doing so, we note that an average of about seven Level change SMe 349 0.82 418 
new teachers were certified in CS each month 

during the time after WeTeach_CS began Slope change 646 044 1459 .000 
compared to an average of less than one new 

teacher per month during the time before Constant 57.47 491 1148 .000 


WeTeach_CS was implemented. Figure 1 
illustrates these trends. 


To check whether these trends were similar or dissimilar to the trends for urban-suburban school districts, we 
conducted the same analysis for CS teachers in those areas. The statistical significance of this analysis was similar to 
that of rural schools in that there was no level change but there was a significant change in slope from the pre- 
intervention to the intervention phase (#(57)=9.11, p<.001), indicating that there was a significant increase in the rate at 
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which teachers in urban-suburban 
districts became certified after the 
start of WeTeach_CS compared to 
the period of time before the 
program. However, the rate at which 
teachers became certified in rural 
areas was substantially larger than 
urban-suburban areas. Whereas the 
number of certified CS teachers in 
urban-suburban areas increased by 
88% during the intervention phase, 
it increased by 169% in rural areas 
during the same period (see Figure 
2). 


To ensure this difference between 
rural and urban-suburban areas was 
unique to the intervention phase and 
not simply a continuation of 
differences that existed prior to the 
intervention, we calculated the 
percent change for comparable 
periods of time before and after the 
start of the WeTeach_CS program. 
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Figure 1. The number of certified CS teachers in rural areas before and after the start of the 
WeTeach_CS program 


Because the intervention phase comprised a two-year period and the pre-intervention phase comprised a period of four 
years in the analysis, we calculated the pre-intervention percent change using just the two years prior to the start of 
WeTeach_CS and found that the number of CS teachers increased by 31% in urban-suburban areas and 34% in rural 
areas during this time period. Thus, whereas the rate in which teachers became certified in urban-suburban areas was 
twice as large during the two years of the intervention phase than the last two years of pre-intervention phase (63% vs 


31%, respectively), it was over five 
times greater in rural areas (178% vs 
34%). 


To aid in ruling out the presence of 
unknown confounding factors, an 
interrupted time series analysis was 
run on the comparison group, rural 
teachers obtaining certification in 
technology applications. In 
conducting this analysis, we followed 
the same procedures used to conduct 
the analysis for rural CS teachers. 
Results showed no statistically 
significance change in level or slope 
(see Table 2). 


Thus, we conclude that there were no 
changes in the number of certified 
technology applications teachers 
immediately after the start of the 
intervention phase that could not be 
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Figure 2. The number of certified CS teachers in rural and urban-suburban areas 
before and after the start of the WeTeach_CS program 
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accounted for by the overall trend, nor was Table 2. Results of the interrupted time series analysis for 


there any difference between the trends in teachers certified in technology applications 


the intervention and pre-intervention B SE t Pp 
puases Pre-intervention slope 433 063 683 .000 

Level change 2.24 1155 -019 847 
Policy Recommendations Slope change 1.27 1.73 0.73 467 
The results of this study demonstrate that Constant 1060.44 20.70 51.24 000 


large-scale collective impact interventions 

can be effective in increasing the number of 

certified CS teachers. Moreover, given the statistically significant change in slope between the pre-intervention and 
intervention phases, this study shows that program effects can be sustained over time. Perhaps most importantly, the 
WeTeach_CS program seemed to have had an even greater impact in rural schools, where the number of certified CS 
teachers increased by 178% compared to 63% in urban-suburban areas. 


Whereas previous research on increasing the teacher workforce in high-need areas focused on interventions enacted on 
relatively smaller scales (typically at the district level), this study provides evidence for the effectiveness of a large- 
scale intervention as it was implemented across an entire state. WeTeach_CS appears to be implementing a promising 
strategy yielding positive results across the large socioeconomic and geographic diversity of Texas. We believe the 
distinguishing factors that have contributed to the success of the WeTeach_CS program lay in its utilization of the 
collective impact model as well as its combination of financial incentives with ample available training. The results of 
this study provide strong evidence for the value of collective impact models for effecting change in rural areas and 
across large geographic regions generally. 


The unique impact on rural areas may have been due to three challenges that rural school districts face to a greater 
degree than urban and suburban districts. First, while many school districts, regardless of geography, lack computer 
science expertise, school districts in urban and suburban areas can often tap into nearby institutions of higher education 
and industry partners to access resources. Second, administrators and teachers in rural communities often have 
overlapping responsibilities due to the small school size and simply lack the bandwidth to launch and support new 
initiatives in which they lack personal expertise. Finally, it if often the case that rural school districts simply lack the 
ability to afford expensive initiatives, such as implementing new courses and training teachers in new content areas. 


We theorize that WeTeach_CS has been more successful in rural communities because it addresses these specific 
challenges. Collective impact models like WeTeach_CS, which function at a large scale and create collaborative 
networks of like-minded organizations, can be especially beneficial for educational institutions in remote areas. With 
the collective impact model, small rural schools can leverage the expertise of the network to compensate for limited CS 
expertise within their organizations. While these schools are geographically isolated, through collective impact they 
become connected to institutions of higher education, industry partners, and other teachers and schools all focused on 
addressing the same challenges. Because rural schools have relatively few teachers, it is difficult for individual school 
districts to achieve an economy of scale to support the creation and growth a CS program. However, in a collective 
impact model the investment can be distributed across an entire statewide network, thus achieving the economy of 
scale that would be too inefficient for rural districts to do on their own. The CS certification preparation training 
provided by the WeTeach_CS program purposefully leverages the strengths of the collaborative network, allowing the 
local organizations to utilize their existing relationships with teachers to propagate the online course and promote in- 
person training sessions. This allowed the initiative to scale more quickly into all areas of the state while keeping the 
financial cost to schools and districts at manageable levels. 


The full published version of this study can be found at https://doi.org/10.1145/3287324.3287418 
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